Systems and methods for guiding and measuring neck and shoulder protraction and retraction motions in virtual/augmented reality

ABSTRACT

A virtual/augmented reality-based system for guiding patients through rehabilitation exercises and providing real time feedback is disclosed. In various embodiments the system, methods, and computer program products relate to guiding and measuring neck and shoulder motion, specifically protraction and retraction motions. A virtual reality environment includes a fixed object having a guidance feature and a moveable object having a complementary shape to the guidance features. Whether the fixed object is in a complimentary orientation to the movable object is determined. When the fixed object is in a complimentary orientation with the movable shape, an indication is presented to the user to perform a motion. A plurality of sensors determines a plurality of measurements relating to the motion of the user. Whether the fixed object is in a complementary orientation with the movable object is determined based on the plurality of measurements.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 62/695,642, filed Jul. 9, 2018, the contents ofwhich are incorporated herein by reference in its entirety.

BACKGROUND

Embodiments of the present disclosure generally relate to guiding andmeasuring neck and shoulder motion, specifically protraction andretraction motions, in virtual reality (VR) or augmented reality (AR)environments.

BRIEF SUMMARY

Systems, methods, and computer program products of the present inventionproviding a virtual environment to the user via a virtual or augmentedreality system. A method of the present disclosure includes providing avirtual reality environment providing a virtual environment to the uservia the virtual or augmented reality system. The virtual realityenvironment includes a first 3D object fixed in space and the first 3Dobject has a first shape. The virtual reality environment includes asecond 3D object fixed to the user and the second 3D object having asecond shape that is complementary to the first shape. Whether the firstshape is in a complimentary orientation to the second shape isdetermined. When the first shape is in a complimentary orientation withthe second shape, the user is indicated to perform a motion via thevirtual or augmented reality system. A plurality of sensors determine aplurality of measurements relating to the motion of the user. Whetherthe second 3D object is in the complementary orientation with the first3D object is determined based on the plurality of measurements.

A system of the present disclosure includes a virtual or augmentedreality display adapted to display a virtual environment to a user, aplurality of sensors coupled to the user, and a computing node includinga computer readable storage medium having program instructions embodiedtherewith. The program instructions are executable by a processor of thecomputing node to cause the processor to perform a method includingproviding a virtual reality environment providing a virtual environmentto the user via the virtual or augmented reality system. The virtualreality environment includes a first 3D object fixed in space and thefirst 3D object has a first shape. The virtual reality environmentincludes a second 3D object fixed to the user and the second 3D objecthaving a second shape that is complementary to the first shape. Whetherthe first shape is in a complimentary orientation to the second shape isdetermined. When the first shape is in a complimentary orientation withthe second shape, the user is indicated to perform a motion via thevirtual or augmented reality system. A plurality of sensors determine aplurality of measurements relating to the motion of the user. Whetherthe second 3D object is in the complementary orientation with the first3D object is determined based on the plurality of measurements.

A computer program product of the present disclosure includes a computerreadable storage medium having program instructions embodied therewith.The program instructions are executable by a processor to cause theprocessor to perform a method including providing a virtual realityenvironment providing a virtual environment to the user via the virtualor augmented reality system. The virtual reality environment includes afirst 3D object fixed in space and the first 3D object has a firstshape. The virtual reality environment includes a second 3D object fixedto the user and the second 3D object having a second shape that iscomplementary to the first shape. Whether the first shape is in acomplimentary orientation to the second shape is determined. When thefirst shape is in a complimentary orientation with the second shape, theuser is indicated to perform a motion via the virtual or augmentedreality system. A plurality of sensors determine a plurality ofmeasurements relating to the motion of the user. Whether the second 3Dobject is in the complementary orientation with the first 3D object isdetermined based on the plurality of measurements.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary virtual reality headset according toembodiments of the present disclosure.

FIG. 2 illustrates an exemplary VR environment having fixed objects andmoveable objects for guiding a patient/user through an exercise andproviding real time feedback to the patient/user according toembodiments of the present disclosure.

FIG. 3 illustrates an exemplary VR environment having a fixed object anda moveable object for guiding a patient/user through an exercise andproviding real time feedback to the patient/user according toembodiments of the present disclosure.

FIG. 4 illustrates an exemplary VR environment having a fixed object anda moveable object for guiding a patient/user through an exercise andproviding real time feedback to the patient/user according toembodiments of the present disclosure.

FIG. 5 illustrates an exemplary VR environment having a fixed object andmoveable objects for guiding a patient/user through an exercise andproviding real time feedback to the patient/user according toembodiments of the present disclosure.

FIG. 6 illustrates an exemplary VR environment having a fixed object andmoveable objects for guiding a patient/user through an exercise andproviding real time feedback to the patient/user according toembodiments of the present disclosure.

FIG. 7 is a flow chart illustrating an exemplary method for guiding apatient/user through an exercise and providing real time feedback to thepatient/user.

FIG. 8 depicts an exemplary computing node according to embodiments ofthe present disclosure.

DETAILED DESCRIPTION

In physical medicine and rehabilitation, also called physical therapy orphysiotherapy, patients work with a physical therapist to enhance and/orrestore their functional ability (and quality of life) after sufferingphysical impairment or disability. In general, a program of physicaltherapy is based on an individual's history and the results of aphysical examination to arrive at a diagnosis. A given physical therapyprogram may integrate assistance with specific exercises, manual therapyand manipulation, mechanical devices such as traction, education,physical agents such as heat, cold, electricity, sound waves, radiation,assistive devices, prostheses, orthoses and other interventions.Physical therapy may also be prescribed as a preventative measure toprevent the loss of mobility before it occurs by developing fitness andwellness-oriented programs for healthier and more active lifestyles.This may include providing therapeutic treatment where movement andfunction are threatened by aging, injury, disease or environmentalfactors. During physical therapy, patients may perform a number ofexercises focused on rehabilitating a particular injured area of thebody (e.g., the neck and/or limbs after a spinal injury) or part of thebody that has undergone a surgical procedure (e.g., carpal tunnelsurgery, knee replacement surgery, shoulder replacement surgery, hipreplacement surgery, muscle/ligament repair, spinal fusion, etc.).

As an example, individuals suffer from neck pain or need to perform neckexercises for various reasons. For example, people who have beeninvolved in a motor vehicle accident or have suffered an injury whileplaying contact sports are prone to develop a whiplash associateddisorder (WAD), a condition resulting from cervicalacceleration-deceleration (CAD). It will be appreciated that this isjust one of many potential injuries that may result in neck injury orpain necessitating rehabilitation.

The majority of people who suffer from non-specific neck pain (NSNP) mayhave experienced symptoms associated with WAD or have an undiagnosedcervical herniated disc. For this population, the recommended treatmentregimen often includes a variety of exercises promoting neck movementand other functional activity training, leading to improvedrehabilitation.

For patients that may not have access to nearby physical therapy, hometraining may have limitations. With no direct guidance from theclinician, the patient has no immediate feedback to confirm correctperformance of required exercises. Lack of such guidance and supervisionoften leads to even lower adherence. As a result, the pain of an initialsensed condition may persist or even worsen—leading to other requiredmedical interventions that could have been prevented, thus alsoincreasing associated costs of the initial condition.

A physical therapist generally has to demonstrate the desired motionsfor each exercise to a patient undergoing the therapy and also correctthe patient if the patient deviates from the proper exercise motion. Fora patient to receive real-time feedback on the exercises, the therapistmust be present to ensure the exercises are performed correctly andprovide the feedback. Improperly performing rehabilitation exercises aredangerous to a patient because the patient may not heal correctly or mayfurther damage the injured area.

Moreover, there is no bio-feedback loop for a patient undergoingrehabilitation therapy, that is, there is no way for the patient toautomatically visualize his/her motion and to correct the motion if themotion is wrong.

Accordingly, a need exists for a system and method that providesreal-time guidance and feedback regarding the accuracy and quality ofphysical therapy exercise motions (e.g., protraction and retraction),and can measure the length of the motion in real time thereby enablingbio-feedback, quantification, and progress control of the rehabilitationprocess.

The VR/AR technology according to various embodiments provides a fullyimmersive environment that enables measurement and guidance of motionperformed by a patient/user. For example, the VR/AR technology may allowmeasurement and guidance of protraction and retraction motions of theneck and shoulders. Within this VR/AR environment, one or more virtualthree-dimensional fixed objects (e.g., one, two three, four, five, six,etc.) are provided such that the fixed object(s) are perceived as havinglength, width, and depth. Each of the three-dimensional fixed objectsmay have a particular defined shape (e.g., box, disk, sphere, cylinder)or may not have any defined shape (e.g., an amorphous shape). Each ofthe fixed objects may include one or more guidance features configuredto guide the motion of the patient/user as will be described in moredetail below. The guidance features may be, for example, a longitudinalcylindrical groove extending along a surface of the fixed object. Theguidance feature may also include one or more holes in the fixed objectand/or one or more targets on the fixed object.

The VR/AR environment also includes one or more virtualthree-dimensional moveable objects that are each associated with a bodypart of the patient/user for motion tracking. In various embodiments,the moveable object in the VR/AR environment is linked to a device thatis attached to the patient's/user's body for which motion is to betracked. The device may interact with the VR/AR technology (e.g., withsensors) by providing positional information to the VR/AR technology formoving the moveable objects in the VR/AR environment. In variousembodiments, each of the moveable objects is associated with, e.g., oneor both shoulders, the neck, one or both arms, one or both legs, and/orany other suitable body part. In various embodiments, the moveableobjects have a particular shape in the VR/AR environment that iscomplementary to the one or more guidance features of the fixedobject(s) described above.

As used herein, complementary, a complementary relationship, orcomplementary orientation (collectively, “complementary”) is where oneshape (e.g., the fixed object) has a groove that corresponds to at leasta part of another shape (e.g., the moveable object) to assist in theguidance of a user's motion during a rehabilitation exercise. In variousembodiments, the fixed object may have a groove that is complementary toa part of the moveable object. In various embodiments, the moveableobject may include a groove that is complementary to a part of the fixedobject.

In an example, the fixed object may be a rectangular block having aguidance feature in the form of a semi-cylindrical groove extendinglongitudinally along the length of a surface while the moveable objectassociated with a shoulder of the patient/user is a spherical shape thatis complementary to the semi-cylindrical shape of the groove. In variousembodiments, the guidance feature may include a convex and/or concavecurvature. In various embodiments, the curvature of the fixed object mayapproximate (e.g., be equal to) the curvature of the movable object. Invarious embodiments, the movable object may include a convex and/orconcave curvature that is complementary to the shape of the guidancefeature. In this particular example, the spherical shape may havesubstantially the same radius as the radius of the semi-cylindricalgroove thereby allowing the spherical shape to fit closely within thegroove or otherwise match the shape of the groove. In other embodiments,the guidance feature may be a rectangular/box shape, triangular shape,pyramidal shape, conical shape, spherical shape, cylindrical shape, orany other suitable shape. In other embodiments, the guidance feature maybe an embossed section instead of a grooved/engraved section. In otherembodiments, the guidance feature need not be a groove or embossed asthe fixed object itself may be a guidance feature (e.g., a disk-shapedtarget).

To initiate a rehabilitation session with a patient/user, the VR/ARtechnology may instruct the patient/user to position the one or moremoveable objects in a complementary relationship with the one or morefixed objects. In the example above, the VR/AR technology may instruct apatient/user to position the movable object (e.g., spherical ball) in acomplementary relationship with the guidance feature (e.g., longitudinalgroove) of the fixed object. The VR/AR technology may instruct thepatient/user to perform a sliding motion while keeping the sphericalball adjacent to the groove. In other embodiments, the VR/AR technologymay instruct a user to motion towards a target.

The VR/AR technology may measure the length of protraction and/orretraction of the patient's/user's body in real time using positionalcoordinates of the moveable object determined from sensors in the VR/ARsystem. Sensors used in VR/AR are described in more detail below. Usingthe above example, the sensors may determine whether the spherical ballis aligned with the groove or if the patient/user positions thespherical ball away from the groove. In the event the spherical ballmoves away from or out of the groove, the VR/AR technology may providefeedback to the user (e.g., through a textual or symbolic warning) thatthe motion is incorrect and provide instruction as to how to correct themotion (e.g., by placing the spherical ball back into the groove).

The VR/AR technology may provide instruction through the form ofdirectional arrows in the VR/AR environment. The directional arrows mayinstruct a patient/user to move in a particular direction or may provideactive feedback for performing a particular motion or exercise. Invarious embodiments, the VR/AR environment may include a feedbackmechanism in the form of arrows. For example, the VR/AR environment mayinclude two feedback arrows (e.g., up and down), four feedback arrows(up, down, left, and right), or six feedback arrows (up, down, left,right, forward, and backward) for providing feedback regarding themotion of the patient/user. Using the above example, if the patient/usershould move the virtual spherical ball up to properly execute anexercise, the up arrow would highlight while the other arrow(s) wouldnot be highlighted. In another example, if the patient/user isperforming the exercise correctly, no arrow may be highlighted and/or anindication of compliance may be shown to the user. In other embodiments,certain features/areas in the VR/AR environment may be highlighted or acolor may be changed to indicate that the motion is incorrect. In theseembodiments, the patient may be instructed to move in the direction thatreduces the highlighted/colored area.

This VR/AR platform thus enables accurate measurements of patient/usermotion during therapy sessions and real-time feedback regardingcompliance with the motions of a particular therapeutic exercise.

It will be appreciated that a variety of virtual and augmented realitydevices are known in the art. For example, various head-mounted displaysproviding either immersive video or video overlays are provided byvarious vendors. Some such devices integrate a smart phone within aheadset, the smart phone providing computing and wireless communicationresources for each virtual or augmented reality application. Some suchdevices connect via wired or wireless connection to an externalcomputing node such as a personal computer. Yet other devices mayinclude an integrated computing node, providing some or all of thecomputing and connectivity required for a given application.

Virtual or augmented reality displays may be coupled with a variety ofmotion sensors in order to track a user's motion within a virtualenvironment. Such motion tracking may be used to navigate within avirtual environment, to manipulate a user's avatar in the virtualenvironment, or to interact with other objects in the virtualenvironment. In some devices that integrate a smartphone, head trackingmay be provided by sensors integrated in the smartphone, such as anorientation sensor, gyroscope, accelerometer, or geomagnetic fieldsensor. Sensors may be integrated in a headset, or may be held by auser, or attached to various body parts to provide detailed informationon user positioning.

In various embodiments, a user is furnished with a VR or AR system. Asnoted above, a VR or AR system will generally have integrated motionsensors. In addition, additional motions sensors may be provided, forexample to be handheld. This allows tracking of multiple patientattributes while they interact with a scene. In this way, systematic andreproducible scenarios may be used to assess the subject's function.

In various embodiments, the guidance feature of the fixed object maydefine a predetermined path. In various embodiments, the predeterminedpath may be selected based on a particular rehabilitation protocol. Forexample, a rehabilitation protocol for neck protraction and/orretraction may involve different motions (and thus have differentpredetermined paths) than a rehabilitation protocol for shoulderprotraction and/or retraction. In various embodiments, the VR/AR systemmay compare a path defined by a plurality of recorded three-dimensionalpoints of the motion of the movable object to the predetermined path ofthe rehabilitation protocol. In various embodiments, the system mayindicate to the user whether or not they are complying with therehabilitation protocol. For example, the VR/AR system may present awarning if the movable object is not in a complementary relationshipwith the guidance feature of the fixed object. In another example, theVR/AR system may determine whether the user followed the entirepredetermined path of the rehabilitation protocol. In this example, theVR/AR system may instruct the user that the rehabilitation protocol isnot complete when the user did not follow the entire path or mayinstruct the user that the rehabilitation protocol is complete if thepredetermined path was followed (while the movable object is in acomplementary orientation with the guidance feature).

In various embodiments, the VR/AR system may instruct a user to hold themovable object in a complementary relationship for a predeterminedamount of time (which may be based on a rehabilitation protocol). Invarious embodiments, the VR/AR system may record the amount of time thatthe movable object is held in the particular position. In variousembodiments, a difference is calculated between the predetermined amountof time and the recorded time. In various embodiments, the VR/AR systemmay notify the user of the elapsed time via, for example, a timer and/ora clock. In various embodiments, if the user does not hold the movableobject in the particular position for at least the predetermined amountof time, the VR/AR system may notify the user that they did not completethe rehabilitation protocol. In various embodiments, if the user holdsthe movable object in the particular position for at least thepredetermined amount of time, the VR/AR system may notify the user thatthey completed the rehabilitation protocol. In various embodiments,results of each rehabilitation session, exercises completed and/or notcompleted, and/or elapsed times, may be recorded in the particularpatient's record within an EHR database.

With reference now to FIG. 1, an exemplary virtual reality headset isillustrated according to embodiments of the present disclosure. Invarious embodiments, system 100 is used to collected data from motionsensors including hand sensors (not pictured), sensors included inheadset 101, and additional sensors such as sensors placed on the body(e.g., torso, limbs, etc.) or a stereo camera. In some embodiments, datafrom these sensors is collected at a rate of up to about 150 Hz. Asillustrated, data may be collected in six degrees of freedom: X axistranslation—left/right; Y axis translation—up/down/height; Z axistranslation forward/backward; P—pitch; R—roll; Y—yaw. As set out herein,this data may be used to track a user's overall motion to facilitateinteraction with a virtual environment and to evaluate theirperformance. Pitch/Roll/Yaw may be calculated in Euler angles.

FIG. 2 illustrates an exemplary VR environment 200 having fixed objects202 a-202 d and moveable objects 204 a-204 d for guiding a patient/userthrough an exercise and providing real time feedback to thepatient/user. In particular, the fixed objects 202 a-202 d are generallyshaped as four rectangular blocks having guidance features 206 a-206 don a surface in the form of semi-cylindrical cut-outs. In variousembodiments, the fixed objects may have any suitable three-dimensionalshape. For example, a fixed object may have a “puzzle piece” shape thatfits into a corresponding moveable object in a specific way. Themoveable objects 204 a-204 d are generally shaped as spheres havingsubstantially the same radius as the guidance features 206 a-206 d ofthe fixed objects 202 a-202 d. The VR environment 200 further includes afeedback mechanism 208 to thereby provide real time feedback to thepatient/user. The feedback mechanism 208 includes a forward arrow 209 aand a backward arrow 209 b that may be used by the VR environment 200 toindicate to the patient/user a specific direction/motion in which tomove, e.g., protraction or retraction. In various embodiments, thefeedback mechanism 208 may indicate the direction that the patientshould move their body.

In an example, the VR environment 200 may highlight the forward arrow209 a of the feedback mechanism if the patient/user is to move theirbody in a forward direction so that a moveable object 204 a-204 dengages the corresponding fixed object 202 a-202 d in the guidancefeature 206 a-206 d as described above. In various embodiments,highlighting may be achieved by coloring the forward arrow 209 a orbackward arrow 209 b. In various embodiments, highlighting may beperformed by increasing the size of the particular arrow and/ordecreasing the size of the other arrow.

In various embodiments, the VR environment 200 may include a target 210to aid the patient in performing a motion. For example, the patient maybe instructed to aim at the target 210 to increase the likelihood thatthe patient performs the motion correctly. In various embodiments, thetarget 210 may be used to measure the rate of accuracy of the usermotion where the user is instructed to aim at the target 210 (or aspecific portion of the target, e.g., bullseye). Measurements of theaccuracy of the patient motion may be used to determine the quality ofthe motions performed. For example, a patient who hits the bullseye ofthe target 210 may have a high accuracy score indicating high qualitymotions.

In various embodiments, two or more (e.g., all four) movable objects 204a-204 d may be associated with a single body part of the user (e.g., theneck, shoulder, hand, foot, etc.). In this embodiment, the user may berequired to align the movable objects 204 a-204 d with the respectiveguidance features 206 a-206 d of the fixed objects 202 a-202 d toinitiate the rehabilitation session. In various embodiments, real-timefeedback may be delivered to the user via the feedback mechanism 208(e.g., up, down, left, and/or right arrows). In various embodiments, thereal-time feedback may include instructions to complete therehabilitation protocol, for example, to move the movable objects 204a-204 d forward towards the target 210. In various embodiments, thereal-time feedback may include a direction to move the one or moremovable objects 204 a-204 d to put the one or more movable objects 204a-204 d in a complementary orientation with the guidance features 206a-206 d.

FIG. 3 illustrates an exemplary VR environment 300 having a fixed object302 and a moveable object 304 for guiding a patient/user through anexercise and providing real time feedback to the patient/user accordingto embodiments of the present disclosure. In this embodiment, the fixedobject 302 is shaped as a beam having a guidance feature 306 in the formof a cubic cut-out along the length of the beam thereby creating aU-shaped beam. The moveable object 304 is generally shaped as a cubehaving substantially the same side length as the guidance feature 306 ofthe fixed object 302 (e.g., the movable object 304 has a complementaryshape to the cubic cut-out). Similar to the VR environment in FIG. 2,the VR environment 300 further includes a feedback mechanism 308 tothereby provide real time feedback to the patient/user. The feedbackmechanism 308 includes a left arrow 309 a and a right arrow 309 b thatmay be used by the VR environment 300 to indicate to the patient/user aspecific direction/motion in which to move a body part (e.g., hand,foot, neck, shoulder, etc.) associated with the movable object 304. InFIG. 3, the left arrow 309 a is highlighted (while right arrow 309 b isnot highlighted) indicating that the user should move the movable object304 to the left once the movable object 304 is placed in a complementaryorientation with the fixed object 302.

As shown in FIG. 3, because the movable object 304 (corresponding to abody part of the user) is not in a complementary orientation with theguidance feature 306 of the fixed object 302, an indication 312 ispresented to the user, e.g., to inform the user that they are not incompliance with a rehabilitation protocol. In particular, the indication312 is a message notifying the user that the movable object 304 is notin a complimentary orientation with the guidance feature 306. In variousembodiments, any suitable indication may be implemented to notify theuser that they are not complying with a rehabilitation protocol, suchas, for example, one or more colors (red, yellow, and/or green), emojis,highlighting, and/or text overlay.

In various embodiments, the one or more arrows 309 a, 309 b may indicatea particular direction of motion of the user to thereby provide the userwith visual feedback of the direction that the movable object istraveling in the VR environment. For example, when the user begins tomove the movable object 304 to the left, the arrow 309 a will behighlighted (and any highlighting on arrow 309 b will disappear). Inthis example, when the user begins to move the movable object 304 to theright, the arrow 309 b will be highlighted (and any highlighting onarrow 309 a will disappear).

In various embodiments, the guidance feature 306 of the fixed object 302defines a path 314 in which the movable object 304 may travel when in acomplementary orientation with the guidance feature 306. In variousembodiments, a rehabilitation protocol may include a predetermined pathprovided to the VR environment via, e.g., an EHR database. In variousembodiments, the VR environment may collect three-dimensional positionaldata points of the movable object that define a path and determine adifference between the path traveled by the movable object and thepredetermined path to assess compliance with the rehabilitationprotocol. In various embodiments, if the difference between the path ofthe movable object 304 and the predetermined path is greater than orequal to a predetermined value, the VR environment 300 may indicate tothe user that they are not in compliance with the rehabilitationprotocol. In various embodiments, if the difference between the path ofthe movable object 304 and the predetermined path is less than apredetermined value, the VR environment 300 may indicate to the userthat they are in compliance with the rehabilitation protocol and/or thatthe rehabilitation protocol is complete.

FIG. 4 illustrates an exemplary VR environment 400 having a fixed object402 and a moveable object 404 for guiding a patient/user through anexercise and providing real time feedback to the patient/user accordingto embodiments of the present disclosure. In this embodiment, the fixedobject 402 is shaped as a winding object having multiple curves 414 a,414 b. In various embodiments, the fixed object 402 may include a singlecurve. In this embodiment, the movable object 404 is spherical where thediameter substantially matches the dimensions of the guidance feature406 of the fixed object 402. Similar to the VR environments describedabove, the VR environment 400 further includes a feedback mechanism 408to thereby provide real time feedback to the patient/user. The feedbackmechanism 408 includes arrows 409 a, 409 b, and 409 c along the guidancefeature 406 that may be used by the VR environment 400 to indicate tothe patient/user a specific direction/motion in which to move a bodypart (e.g., hand, foot, neck, shoulder, etc.) associated with themovable object 404. In FIG. 4, the first arrow 409 a closest to thestarting point is highlighted (while arrows 409 b and 409 c are nothighlighted) indicating that the user should move the movable object 404along the winding guidance feature 406 once the movable object 404 isplaced in a complementary orientation with the fixed object 402.

In various embodiments, as the user moves the movable object 404 alongthe curved guidance feature 406 of the fixed object 402, the next arrow409 b may be highlighted. For example, when the first arrow 409 b ispassed by the movable object 404 (due to the motion of the user whilethe movable object is in a complementary orientation with the guidancefeature of the fixed object), the highlighting of the first arrow 409 amay disappear and the second arrow 409 b may be highlighted. Similarly,when the second arrow 409 b is passed, the highlighting on the secondarrow 409 b may disappear and the third arrow 409 c may be highlighted.In various embodiments, the arrows 409 a-409 c may maintain thehighlighting as the movable object 404 passes each arrow.

FIG. 5 illustrates an exemplary VR environment 500 having a fixed object502 and moveable objects 504 a-504 c for guiding a patient/user throughan exercise and providing real time feedback to the patient/useraccording to embodiments of the present disclosure. In this embodiment,the VR environment 500 includes guidance features 506 a-506 c thatspecifically correspond to a particular movable object 504 a-504 c. Forexample, the guidance feature 506 a is square in shape and therefor hasa complementary orientation to the cubic movable object 504 a.Similarly, the guidance feature 506 b is circular in shape and thereforhas a complementary orientation to the spherical movable object 504 bwhile the guidance feature 506 c is triangular in shape and therefor hasa complementary orientation to the triangular movable object 504 c.

Similar to the embodiments above, the VR environment 500 includes afeedback mechanism 508 that includes three arrows 509 a-509 c. Invarious embodiments, each arrow 509 a-509 c may be used to indicate tothe patient which movable object 504 a-504 c to put in a complementaryorientation with its respective guidance feature 506 a-506 c. Forexample, in FIG. 5, the first arrow 509 a is highlighted therebyindicating to the user that the first movable object 509 a should beplaced in a complementary orientation with the square-shaped guidancefeature 506 a. In this example, the other two arrows 509 b, 509 c arenot highlighted while the first arrow 509 a is highlighted. In variousembodiments, two or more arrows 509 a-509 c may be highlighted at thesame time. In various embodiments, the arrows 509 a-509 c may behighlighted one at a time (e.g., sequentially) after a predeterminedamount of time.

In various embodiments, the user may be instructed to hold one or moreof the movable objects 509 a-509 c in a complementary orientation withthe respective guidance feature(s) 506 a-506 c for a predeterminedamount of time. In various embodiments, the VR environment may recordthe amount of time that the user holds the movable object 504 a-504 c inthe complementary orientation with its respective guidance feature 506a-506 c. In various embodiments, the VR environment 500 may present theuser with a timer and/or clock so that the user may be aware of how longthe movable object 504 a-504 c should be held in the complementaryorientation. In various embodiments, the VR environment 500 maydetermine a compliance metric based on the recorded time. In variousembodiments, the recorded time may be compared to the predetermined timeto determine a difference. In various embodiments, if the difference isgreater than a predetermined threshold, the user may be notified thatthe rehabilitation protocol is not complete. In various embodiments, ifthe difference is less than a predetermined threshold, the user may benotified that the rehabilitation protocol is complete.

FIG. 6 illustrates an exemplary VR environment 600 having a fixed object602 and moveable objects 604 a, 604 b for guiding a patient/user throughan exercise and providing real time feedback to the patient/useraccording to embodiments of the present disclosure. The fixed object 602includes a guidance feature 606 a that is a slot with arcuate ends and aguidance feature 606 b that is a slot with square ends. Due to thearcuate shape of the ends of the guidance feature 606 a, the firstmovable object 604 a is capable of a complementary relationship with theguidance feature 606 a throughout the entire length of the guidancefeature 606 a. Similarly, due to the square shape of the ends of theguidance feature 606 b, the second movable object 604 b is capable of acomplementary relationship with the guidance feature 606 b throughoutthe entire length of the guidance feature 606 b.

The VR environment 600 further includes a first feedback mechanism 608 aassociated with the first movable object 604 a and a second feedbackmechanism 608 b associated with the second movable object 604 b. Thefirst feedback mechanism 608 a includes arrows 609 a, 609 b and thesecond feedback mechanism 608 b includes arrows 609 c, 609 d. In FIG. 6,arrow 609 a is highlighted to indicate to the user to move the firstmovable object 604 a in the upward direction while the movable object604 a is in a complementary relationship with the guidance feature 606a. In various embodiments, an arrow from both feedback mechanisms 608 a,608 b may be highlighted at the same time to indicate to the user thatboth movable objects 604 a, 604 b should be put in a complementaryrelationship with the respective guidance features 606 a, 606 b andmoved in the indicated direction. In various embodiments, based on therehabilitation protocol, the user may be instructed to move one or moreof the movable objects 604 a, 604 b in the direction(s) indicated by thefeedback mechanism(s) 608 a, 608 b. In various embodiments, the user maybe instructed to hold one or more movable object 604 a, 604 b at aparticular location within the guidance feature(s) 606 a, 606 b for apredetermined amount of time. The predetermined amount of time may bebased on the particular rehabilitation protocol and may be any suitableamount of time as is known in the art.

FIG. 7 illustrates a flowchart for a method for guiding a patient/userthrough an exercise and providing real time feedback to thepatient/user. At 702, the method includes providing a virtualenvironment to the user via the virtual or augmented reality system. Thevirtual environment includes a first 3D object fixed in space, the first3D object having a first shape, and a second 3D object fixed to theuser, the second 3D object having a second shape that is complementaryto the first shape. At 704, the method includes determining whether thefirst shape is in a complimentary orientation to the second shape. At706, the method includes, when the first shape is in a complimentaryorientation with the second shape, indicating to the user to perform amotion via the virtual or augmented reality system. At 708, the methodincludes determining by a plurality of sensors a plurality ofmeasurements relating to the motion of the user. At 710, the methodincludes determining whether the second 3D object is in thecomplementary orientation with the first 3D object based on theplurality of measurements.

In various embodiments, off the shelf VR systems are optionally usedwith additional external compatible sensors to track various elements inmultiple fields including, e.g., motion tracking, cognitive challenges,speech recognition, stability, facial expression recognition, andbiofeedback.

Motion tracking can include, but is not limited to tracking of gait,stability, tremors, amplitude of motion, speed of motion, range ofmotion, and movement analysis (smoothness, rigidity, etc.).

Cognitive challenges can include, but is not limited to reaction time,success rate in cognitive challenges, task fulfillment according todifferent kind of guidance (verbal, written, illustrated, etc.),understanding instructions, memory challenges, social interaction, andproblem solving.

Speech Recognition can include, but is not limited to fluent speech,ability to imitate, and pronunciation.

Stability can include, but is not limited to postural sway.

Bio-Feedback can include, but is not limited to, Heart rate variability(HRV), Electrothermal activity (EDA), Galvanic skin response (GSR),Electroencephalography (EEG), Electromyography (EMG), Eye tracking,Electrooculography (EOG), Patient's range of motion (ROM), Patient'svelocity performance, Patient's acceleration performance, and Patient'ssmoothness performance.

The following is a non-limiting example of guiding a patient/userthrough an exercise and providing real time feedback to the patient/usein accordance with various embodiments. A user wears a VR/AR headset(with optional tracking device on a particular body part) and enters avirtual environment. The user is provided a series of instructionsand/or demonstrations regarding how to perform an exercise as a part oftheir rehabilitation plan. The user proceeds to perform the exercise andcan visualize the moveable objects and fixed objects in the virtualenvironment used for guiding the exercise. The VR system may track themotion of the user and collect positional data of the user's body. Whileperforming the exercise, the VR system may indicate which direction theuser is to move their body by, for example, highlighting one or morearrows or areas within the three-dimensional object. In variousembodiments, if the user deviates from the proper exercise motion, thevirtual environment may provide an indication to the user that themotion is incorrect and provide real time feedback regarding how tocorrect the motion by, for example, highlighting one or more arrows orareas within the three-dimensional object.

Referring now to FIG. 8, a schematic of an example of a computing nodeis shown. Computing node 10 is only one example of a suitable computingnode and is not intended to suggest any limitation as to the scope ofuse or functionality of embodiments of the invention described herein.Regardless, computing node 10 is capable of being implemented and/orperforming any of the functionality set forth hereinabove.

In computing node 10 there is a computer system/server 12, which isoperational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with computer system/server 12 include, but are notlimited to, personal computer systems, server computer systems, thinclients, thick clients, handheld or laptop devices, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputer systems, mainframecomputer systems, and distributed cloud computing environments thatinclude any of the above systems or devices, and the like.

Computer system/server 12 may be described in the general context ofcomputer system-executable instructions, such as program modules, beingexecuted by a computer system. Generally, program modules may includeroutines, programs, objects, components, logic, data structures, and soon that perform particular tasks or implement particular abstract datatypes. Computer system/server 12 may be practiced in distributed cloudcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed cloud computing environment, program modules may be locatedin both local and remote computer system storage media including memorystorage devices.

As shown in FIG. 8, computer system/server 12 in computing node 10 isshown in the form of a general-purpose computing device. The componentsof computer system/server 12 may include, but are not limited to, one ormore processors or processing units 16, a system memory 28, and a bus 18that couples various system components including system memory 28 toprocessor 16.

Bus 18 represents one or more of any of several types of bus structures,including a memory bus or memory controller, a peripheral bus, anaccelerated graphics port, and a processor or local bus using any of avariety of bus architectures. By way of example, and not limitation,such architectures include Industry Standard Architecture (ISA) bus,Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnect (PCI) bus.

Computer system/server 12 typically includes a variety of computersystem readable media. Such media may be any available media that isaccessible by computer system/server 12, and it includes both volatileand non-volatile media, removable and non-removable media.

System memory 28 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 30 and/or cachememory 32. Computer system/server 12 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 34 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 18 by one or more datamedia interfaces. As will be further depicted and described below,memory 28 may include at least one program product having a set (e.g.,at least one) of program modules that are configured to carry out thefunctions of embodiments of the invention.

Program/utility 40, having a set (at least one) of program modules 42,may be stored in memory 28 by way of example, and not limitation, aswell as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. Program modules 42 generally carry out the functions and/ormethodologies of embodiments of the invention as described herein.

Computer system/server 12 may also communicate with one or more externaldevices 14 such as a keyboard, a pointing device, a display 24, etc.;one or more devices that enable a user to interact with computersystem/server 12; and/or any devices (e.g., network card, modem, etc.)that enable computer system/server 12 to communicate with one or moreother computing devices. Such communication can occur via Input/Output(I/O) interfaces 22. Still yet, computer system/server 12 cancommunicate with one or more networks such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter 20. As depicted, network adapter 20communicates with the other components of computer system/server 12 viabus 18. It should be understood that although not shown, other hardwareand/or software components could be used in conjunction with computersystem/server 12. Examples, include, but are not limited to: microcode,device drivers, redundant processing units, external disk drive arrays,RAID systems, tape drives, and data archival storage systems, etc.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

1. A method comprising: providing a virtual environment to the user viathe virtual or augmented reality system, the virtual environmentcomprising: a first 3D object fixed in space, the first 3D object havinga first shape, and a second 3D object fixed to the user, the second 3Dobject having a second shape that is complementary to the first shape;determining whether the first shape is in a complimentary orientation tothe second shape; when the first shape is in a complimentary orientationwith the second shape, indicating to the user to perform a motion viathe virtual or augmented reality system; determining by a plurality ofsensors a plurality of measurements relating to the motion of the user;and determining whether the second 3D object is in the complementaryorientation with the first 3D object based on the plurality ofmeasurements.
 2. (canceled)
 3. The method of claim 1, wherein the first3D object defines a first path, and wherein the plurality ofmeasurements define a second path.
 4. The method of claim 3, wherein thefirst path and the second path each comprises a plurality ofthree-dimensional points.
 5. The method of claim 3, wherein determiningwhether the second 3D object is in the complementary orientation withthe first 3D object comprises determining a difference between the firstpath and the second path.
 6. The method of claim 5, further comprising,when the difference is greater than a predetermined threshold, providingan indication to the user that the second 3D object is not in acomplementary relationship with the first 3D object.
 7. The method ofclaim 6, further comprising, when the difference is greater than apredetermined threshold, providing an indication to the user of adirection of motion to position the second 3D object back in thecomplementary relationship with the first 3D object.
 8. The method ofclaim 6, further comprising, when the difference is greater than orequal to the predetermined threshold, providing an indication to theuser that the second 3D object is in a complementary relationship withthe first 3D object. 9-18. (canceled)
 19. A system comprising: a virtualor augmented reality display adapted to display a virtual environment toa user; a plurality of sensors coupled to the user; a computing nodecomprising a computer readable storage medium having programinstructions embodied therewith, the program instructions executable bya processor of the computing node to cause the processor to perform amethod comprising: providing a virtual environment to the user via thevirtual or augmented reality system, the virtual environment comprising:a first 3D object fixed in space, the first 3D object having a firstshape, and a second 3D object fixed to the user, the second 3D objecthaving a second shape that is complementary to the first shape;determining whether the first shape is in a complimentary orientation tothe second shape; when the first shape is in a complimentary orientationwith the second shape, indicating to the user to perform a motion viathe virtual or augmented reality system; determining by a plurality ofsensors a plurality of measurements relating to the motion of the user;and determining whether the second 3D object is in the complementaryorientation with the first 3D object based on the plurality ofmeasurements.
 20. (canceled)
 21. The system of claim 19, wherein thefirst 3D object defines a first path, and wherein the plurality ofmeasurements define a second path.
 22. (canceled)
 23. The system ofclaim 21, wherein determining whether the second 3D object is in thecomplementary orientation with the first 3D object comprises determininga difference between the first path and the second path.
 24. The systemof claim 23, further comprising, when the difference is greater than apredetermined threshold, providing an indication to the user that thesecond 3D object is not in a complementary relationship with the first3D object.
 25. The system of claim 24, further comprising, when thedifference is greater than a predetermined threshold, providing anindication to the user of a direction of motion to position the second3D object back in the complementary relationship with the first 3Dobject.
 26. The system of claim 24, further comprising, when thedifference is greater than or equal to the predetermined threshold,providing an indication to the user that the second 3D object is in acomplementary relationship with the first 3D object. 27-36. (canceled)37. A computer program product comprising a computer readable storagemedium having program instructions embodied therewith, the programinstructions executable by a processor to cause the processor to performa method comprising: providing a virtual environment to the user via thevirtual or augmented reality system, the virtual environment comprising:a first 3D object fixed in space, the first 3D object having a firstshape, and a second 3D object fixed to the user, the second 3D objecthaving a second shape that is complementary to the first shape;determining whether the first shape is in a complimentary orientation tothe second shape; when the first shape is in a complimentary orientationwith the second shape, indicating to the user to perform a motion viathe virtual or augmented reality system; determining by a plurality ofsensors a plurality of measurements relating to the motion of the user;and determining whether the second 3D object is in the complementaryorientation with the first 3D object based on the plurality ofmeasurements.
 38. (canceled)
 39. The computer program product of claim37, wherein the first 3D object defines a first path, and wherein theplurality of measurements define a second path.
 40. The computer programproduct of claim 39, wherein the first path and the second path eachcomprises a plurality of three-dimensional points.
 41. The computerprogram product of claim 40, wherein determining whether the second 3Dobject is in the complementary orientation with the first 3D objectcomprises determining a difference between the first path and the secondpath.
 42. The computer program product of claim 41, further comprising,when the difference is greater than a predetermined threshold, providingan indication to the user that the second 3D object is not in acomplementary relationship with the first 3D object.
 43. The computerprogram product of claim 42, further comprising, when the difference isgreater than a predetermined threshold, providing an indication to theuser of a direction of motion to position the second 3D object back inthe complementary relationship with the first 3D object.
 44. Thecomputer program product of claim 42, further comprising, when thedifference is greater than or equal to the predetermined threshold,providing an indication to the user that the second 3D object is in acomplementary relationship with the first 3D object. 45-54. (canceled)